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Team Based Engineering Design Thinking 

The objective of this research was to explore design thinking among teams 
of high school students. This objective was encompassed in the research 
question driving the inquiry: How do teams of high school students allocate time 
across stages of design? Design thinking on the professional level typically 
occurs in a team environment. Many individuals contribute in a variety of ways 
to facilitate the successful development of a solution to a problem. Teachers 
often require students to work in teams, but little is known about how the team 
functions in the context of design and the potential interaction between team 
performance and authentic design challenges. Few research results are available 
to guide high school teachers in developing successful design teams and how to 
encourage teams in their efforts. 

Problem Statement 

The National Center for Engineering and Technology Education has 
formalized a focus on infusing engineering design into secondary technology 
education (National Center for Engineering and Technology Education, 2013). 
Results of the Center’s work have included engaging students in engineering 
design challenges using a problem based learning framework. While extensive 
efforts have be afforded to developing learning experiences and professional 
development activities for teachers, little attention has been given to how teams 
of high school students allocate their design time. The endeavor to model design 
problem solving satisfactorily has engaged scholars across domains (Hayes, 
1989; Newell & Simon, 1972; Polya, 1945; Rubenzer, 1979). Understanding 
how students use their time provides educators with opportunities to improve 
areas where students spend little effort. This study investigates how teams of 
students allocate time in the design process and draws comparisons between two 
different design problems administered. Comparisons are provided between 
teams in this study and previous studies on individual experts and individual 
students. 

The discrepancy between our society’s dependence upon technology and 
our ability to understand various technological issues has emerged as a serious 
concern for educators. “Technology is the outcome of engineering; it is rare that 
science translates directly into technology, just as it is not true that engineering 
is just applied science” (National Academy of Engineering, 2004, p. 7). 
Specifically, “Americans are poorly equipped to recognize, let alone ponder or 
address, the challenges technology poses or the problems it could solve” 
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(Pearson & Young, 2002, pp. 1-2). The relationship between understanding 
engineering and technological literacy is of special urgency during the high 
school years, since “technologically literate people should also know something 
about the engineering design process” (Pearson & Young, 2002, p. 18). 

Design thinking is fundamental to understanding the technologically 
dependent nature of our society. A need for a technologically literate populace, 
therefore, includes an understanding of the engineering design process. The 
design process links technology and engineering, two elements of STEM 
education. “Design is the central component of the practice of engineering and a 
key element in technology education” (Pearson & Young, 2002, p. 58). While 
design thinking is an elusive and difficult construct to define, measurements for 
this study included a pertinent subset of measurements consistent with previous 
literature, much of which was generated through work of the Center for 
Engineering Learning and Teaching (Atman, Chimka, Bursic, & Nachtmann, 
1999; Atman, Kilgore, & McKenna, 2008; Morozov, Yasuhara, Kilgore, & 
Atman, 2008; Mosborg et al., 2005; Mosborg et ah, 2006). This paper reports on 
measurements including time allocated across essential elements of the design 
process and disaggregates the data by problem type and team gender 
composition. 

Methodology 

A descriptive study was conducted spanning multiple high schools in urban, 
rural and suburban environments. This study identified quality high school 
technology and engineering learning and teaching environments in a criterion 
based sampling strategy. Criterion for selective schools was aligned with the 
vision of Pearson and Young, where “technology teachers with a good 
understanding of science and the interactions between technology, science, and 
society will be well prepared to work with other teachers to integrate technology 
with other subjects” (p. 108). Teachers at the target schools permitted 
advertising to recruit their students for participation in the study. Students in this 
study were considered to be representative of experienced students who had 
taken most or all engineering related courses at their high school. Students were 
recruited who were actively engaged in the study of engineering design through 
a criterion sampling strategy (Creswell, 1998) using the following criteria: 

• The high schools had an established program of study which employs a 
focus on engineering in a sequence of courses developed in association 
with an engineering outreach effort as part of a university program. 

• In these courses, students participated in design activities which engage 
their critical thinking and problem solving skills within the framework 
of the engineering design process. 

Sample. The quantitative research method design leveraged the use of data 
from 17 design teams comprised of 2-4 students each for a total of 47 students. 
The 17 teams were composed of eight male only teams, four female only teams, 


- 53 - 



Journal of Technology Education 


Vol. 25 No. 2, Spring 2014 


and five mixed gender teams. The teachers grouped the students into teams 
according to their personal schedules, and the teams were assigned the design 
challenge. Some team members were friends while other teams were comprised 
of students who did not know each other well. All members of a team were from 
the same school, but the 17 teams spanned four schools in two states. Each of 
the schools selected to participate had a recognized engineering program 
associated with an outreach effort by a university engineering program. 
Curricular offerings at the high schools included Project Lead the Way (PLTW), 
Engineering Projects in Community Service (EPICS) High, First Robotics, and 
locally developed engineering and/or technology courses supported by their 
regional University. 

Demographics. Students were selected who represent diverse backgrounds 
and have chosen to enroll in this sequence of courses. Most students in this 
study were seniors who had taken multiple engineering courses during their high 
school experience. Typically participants were enrolled in a senior level 
capstone design course such as the Project Lead the Way, Engineering Design 
and Development course. Approximately one-third of the students were female 
and most reported their ethnicity as White or Asian. About one-half of the 
students responded to a question about their future career choice and nearly 
three-quarters of those indicated engineering. Refer to table 1 for demographics. 


Table 1 

Student Demographics 

Student 

Response 

Mean Number of Engineering Courses 3.6 courses 

Expressed interest in Engineering 68% 

Females 34% 

Seniors 74% 

Underrepresented in Engineering 17% 


Engineering Design Problems. According to the National Center for 
Engineering and Technology Education Caucus Report of 2012, “There is a 
need for more definitive guidance about what makes quality design challenges 
and how they can be implemented well in existing courses” (p. 2). Two different 
design problems were administered in this study which were open-ended, 
realistic, accessible, and complex (Mosborg, et al., 2006). The “playground 
problem” was provided to permit comparisons between team and individual 
performance while the “street crossing problem” variation facilitated 
comparisons among types of problem structures. The “playground problem” was 
comparable to the design problem used in previous studies with individual high 
school students (Mentzer, Becker, & Park, 2011) and previous work with 
college students, and experts (Atman, et al., 1999; Mosborg, et al., 2005; 
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Mosborg, et al., 2006). The street crossing problem was less structured, could be 
readily adapted in order to be locally relevant, and was potentially more 
authentic for the participants because it was situated in a local context 
experienced daily by the students. In addition, students engaged in the 
pedestrian flow problem had an opportunity to specify constraints and criteria as 
none were presented. The “pedestrian flow problem”, administered to 
approximately half of the teams, was a variation of the “street crossing problem” 
adapted from previous literature (Cardella, Atman, Turns, & Adams, 2008; 

Carie Mullins, Atman, & Shuman, 1999). 

Administration of the Design Challenges. Eight of the teams received a 
playground design problem and nine received a locally relevant school hallway 
traffic flow problem. Teams were expected to develop a solution in 2 hours. The 
interactions of group members were video and audio recorded while they were 
developing the design solution. Data included video and audio recordings of the 
design sessions. Video cameras were small, mounted on miniature tripods to 
minimize their intrusion. All students were wired with a lavaliere microphone to 
ensure high quality audio feeds. Wires were run under the team workspace to 
prevent tangling, however, the wires limited student mobility. Students 
generated documents and other artifacts with traditional office supplies 
provided. Artifacts typically included sketches, notes, and formal drawings. 

The Playground Problem has been used in multiple studies and can be 
traced to Dally and Zang (1993). The original need for project driven 
approaches in the freshman engineering design course was to increase student 
performance and retention and to situate student learning of abstract concepts 
through real world applications in an experiential activity. In the original 
activity, students designed a swing set with slides and seesaw. Atman et al. 
(1999) revised the foundational work of Dally and Zang to create a playground 
design problem. In their challenge, engineering students were presented with a 
brief playground design task and access to background information upon 
request. Participants were provided with a maximum of three hours to develop a 
solution to the problem while thinking aloud. Mosborg et al. (2005) applied the 
playground design challenge using the “think aloud” research protocol with 19 
practicing engineers who were identified as experts in the field. Mosborg et al. 
(2006) compared groups of freshman and senior engineering students with 
practicing engineers using data their research team previously collected on the 
playground design challenge. Atman et al. (2008) analyzed data from previous 
studies with a focus on the language of design, its relationship to design thinking 
as a mediator, and relationships between the internalization of design thinking 
and language acquisition. 

Consistent with previous studies on college and high students, participants 
were given a one page design brief of the playground problem. The participating 
teams, acting as engineers, were assigned to design a playground on a donated 
city block. The constraints included limited budget, child safety, and compliance 
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with zoning regulations and applicable laws. Participants were able to query the 
research administrator for additional specific information on the lot layout, cost 
of materials, neighborhood demographics, or other information. There was a 
two-hour time limit for completion of the design proposal, which was a 
modification of the original three hour limit in previous studies (Becker, 
Mentzer, & Park, 2012). This modification was made because the average 
design time in the previous study of high school students was about 90 minutes. 
The two-hour limit provided more time than the average individual needed, yet 
it reduced the resources needed for data collection. The participants presented a 
written proposal describing their design. This activity engaged the participants 
in problem framing and the development of an initial solution. Limitations of 
this design task included the lack of opportunity for participants to investigate 
the need for a solution, since the problem was simply assigned to them. Students 
did not have an opportunity to construct physical models or prototypes. 
Participants were aware that implementation of the design project would not 
occur and that their designs would not be realized. 

The Street Crossing Design Problem was adapted from previous research 
(Cardella, et al., 2008; Carie Mullins, et al., 1999). The National Center for 
Engineering and Technology Education assembled a Caucus in August 2011 to 
identify characteristics of engineering design challenges (Householder, 2011). 
Results of the discussions by this group of experts indicated that excellent 
design challenges should incorporate the following characteristics: 

• Authenticity 

• Have personal and social relevance 

• Require analytical thinking 

• Involve group efforts 

• Require hands-on participation 

• Are clearly structured but open-ended 

• Foster creative solutions 

• Consider ethical issues 

• Meet applicable constraints 

• Provide opportunities for modeling with replication 

• Consider systems implications 

• Are well documented 

• Are self-assessed and independently evaluated 

• Enable communication among team members 

The Street Crossing Design Problem was potentially more authentic and 
more closely aligned with the National Center’s Caucus suggestions than the 
playground problem. “Authentic problems currently affect real-life situations 
encountered by the learners, their families, and their communities - and they do 
not have a generally recognized "right answer.” (National Center for 
Engineering and Technology Education, 2012, p. 22). The problem was 
modified slightly from its original administration to more closely exemplify 
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these characteristics. The street crossing problem was discussed with teachers at 
the schools involved. Teachers were asked to think about an intersection fitting 
characteristics of the original problem but located at or near the school where 
students would immediately recognize the problem as they personally 
experienced it daily. After negotiating with the teachers, it was discovered that 
car/pedestrian traffic flow was an issue, but a more relevant and pervasive 
similar issue was pedestrian (student) flow in school hallways. Each school had 
one or more significant blockages that caused congestion, frustration, and delay 
at passing times between classes. The design problem was modified to focus on 
student hallway flow rather than car traffic flow, making the problem more 
relevant and personal, as most students experience the congestion several times 
per day. 

Each pedestrian flow problem was presented in a similar format: the school 
floor plan (“map”) was provided to participants along with a very brief narrative 
stating that the student team was a team of engineers contacted by the school 
district. The narrative introduced an area that the students immediately 
recognized as a congested area and requested that the team present a proposal 
for resolving the issue. The constraints and criteria were not specified; leaving 
student design teams the opportunity to discuss and negotiate their specific 
problem definition and determine the most appropriate solution proposal. An 
example pedestrian flow problem looked like this, though details varied across 
schools to situate the problem in the local context: 

You are a team of engineer’s contacted by [your school name here] School 
District. Often hallways are congested at passing time between classes. 
Hallway one, which is between the new and old portions of the school, is 
difficult to navigate, [your school name here] School District would like 
your team to develop and propose a solution. 

This less structured problem is consistent with the National Center’s suggestion 
that “Engineering design challenges are ill-structured problems that may be 
approached and resolved using strategies and approaches commonly considered 
to be engineering practices” (2012, p. 2). Typical office supplies were provided 
for the participants, a condition similar to those in the playground problem. 
However, participants were given access to the Internet in lieu of printed sheets 
of relevant data. The decision to provide Internet access was made to increase 
the sense of authenticity and relevance as students are familiar with and 
accustomed to having Internet access. The notion of having predetermined what 
information is needed for their solutions may unintentionally guide student 
design decisions to those based upon a finite resource pool. 

Data Analysis. Time is a limited resource and the ways designers allocate 
their time among the areas of the design process has been a focus of previous 
work. The coding scheme was congruent with the approach used in earlier 
studies (Atman, et al., 1999; Bursic & Atman, 1997; Mosborg, et ah, 2005; 
Mosborg, et ah, 2006). Two measurements of time were made while the 
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designers were at work: time allocated to elements of the design process; and 
total time engaged in design. The unit of analysis was the team. The data were 
coded into the nine categories presented by Mosborg et al. (2006, p. 15): (1) 
Problem Definition, defining what the problem really is; (2) Gather Information, 
searching for and collecting information needed to solve the problem; (3) 
Generating Ideas, thinking up potential solutions (or parts of potential solution) 
to the problem; (4) Modeling, detailing how to build the solution (or parts of the 
solution) to the problem; (5) Feasibility Analysis, assessing and passing 
judgment on a possible or planned solution to the problem; (6) Evaluation, 
comparing and contrasting two (or more) solutions to the problem on a 
particular dimension (or set of dimensions) such as strength or cost; (7) 

Decision, selecting one idea or solution to the problem (or parts of the problem) 
from among those considered; (8) Communication, the participants’ 
communicating elements of the design in writing, or with oral reports, to parties 
such as contractors and the community; and (9) Other, none of the above codes 
apply. Statements coded as other included administrative questions such as 
“should we draw this on paper?” and “can 1 use a calculator?”. Statements coded 
as “other” also were statements that were vague and ambiguous. Statements 
were only coded with one of the eight codes if reasonable evidence existed to 
justify the claim and therefor a statement that was made but not understood by 
the coders was assigned as “other”. 

Data analysis began with segmenting the data sets. A team of three 
researchers was tasked with the responsibility of segmenting. A segment was 
defined as a pause bound utterance, as suggested by Atman et al. (1999). 
Researchers were instructed to create a new segment in the video timeline for 
each instance when any student on the team began a new thought, which was 
typically indicated by beginning to speak after a pause. In previous literature 
(Becker, et al., 2012; Mentzer & Becker, 2010; Mentzer, et al., 2011), this 
segmenting procedure was applied to individuals. For the current study of teams, 
researchers created these segments each time any member of the team made a 
transition. The resulting segmented data represented the composite of all team 
member segments. At some points in the videos, all team members were 
functioning as one cohesive unit and segmenting was simple and monolithic. In 
other times, a team of four students might naturally divide into two teams of two 
and the segments represent start/stop times for each sub-team. By segmenting in 
this fashion, a divergence in design activities could be coded with two separate 
codes in two different, but overlapping episodes. 

Quantitative measures of inter rater reliability on the segmenting process 
were not made. The research leadership determined the segmenting would be of 
reasonable quality if the inter-rater reliability measures for coding were high. If 
segmenting were done successfully, coding could potentially result in high inter 
rater reliability. Coding served as a proxy for quality control of the segmenting 
process. As a preliminary quality control mechanism, the lead researcher 
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reviewed segmented work and provided feedback and guidance as the research 
assistants progressed. 

Two undergraduate students coded the data in three phases. The first phase 
was to establish calibration of the research assistant’s coding work. The second 
phase served to document the calibration using Kappa values as a measure of 
inter-rater reliability. The team of two research assistants coded 25% of each 
video and compared. The Kappa values averaged 0.71; details are presented in 
Table 2 (next page) along with the number of references used to generate the 
values. In the third phase, all videos were fully coded, approximately one-half 
by one research assistant and one-half by the other. 

In the calibration phase, research assistants were provided with a conceptual 
overview of the coding process, structure, technique and rationale. They were 
presented with examples from previous work and practiced coding these data. 
Research assistants then coded a portion of a video and compared with each 
other. They met with a senior research team member and discussed the 
individual interpretations and differences to establish clarification on coding. A 
“Dynamic” Code Book was adopted and maintained. This was a document with 
very specific examples of the different codes developed by creating a 
description of the code and compilation of examples in context. This included 
adding detail and clarifying the meaning of segmenting and coding procedures 
and providing examples as coders did their work. The document was updated 
regularly and shared via network real time. As understanding and interpretation 
was negotiated by the coders and research team leaders, the codebook 
documents evolved into increasingly specific definitions. 

The calibration process was iterative. Each coding session was followed by 
a debriefing session and the cycle started over. Kappa values began relatively 
low and rose gradually as the research assistants became more closely aligned in 
their designations. When average Kappa values for each code approached 0.70, 
the research team transitioned into the next phase which was documentation. 
Some effort was focused on calibration, but most effort was allocated toward 
coding a random 25% of each video and documenting the comparison. 
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Table 2 

Cohen’s Kappa For Each Design Activity 


Design Activity 

Cohen’s Kappa 

References Compared 

Problem Definition 

0.76 

152 

Gathering Information 

0.72 

630 

Generating Ideas 

0.66 

65 

Modeling 

0.68 

1412 

Feasibility 

0.46 

294 

Evaluation 

0.75 

26 

Decision 

0.80 

15 

Communication 

0.88 

732 

Average Inter-Rater Reliability 

0.71 



In the final phase of coding, the 17 videos were divided among the two 
research assistants. Earlier work resulted in 25% of each video being coded 
already; the remaining 75% was coded. The entire video was reviewed and 
changes were made as needed to the coding structure in context of the newly 
coded 75%. 


Results 

The video data were coded by time allocated to: Problem Definition, 
Gathering Information, Generating Ideas, Modeling, Feasibility, Evaluation, 
Decision Making and Communication efforts. Activities that the team engaged 
in were coded. Key differences are discussed in terms of comparisons between 
individuals and teams, problem types and team gender. At times in the process, 
team members were all simultaneously engaged in one activity, but, at other 
times, individual students would engage in different activities. When team 
members provided reasonable evidence that they were doing two different 
activities, two or more codes were applied. The total coded data exceeded 100% 
in all teams because, at times, the team was receiving credit for two or more 
codes simultaneously. With this study’s small sample size (n=17) statistical 
analysis was not conducted. However, trends emergent in the time allocation 
between individuals and teams may provide a foundation for future study. 

Individuals vs Teams. Teams averaged 102 minutes in the design process 
as compared to individuals from previous work who finished, on average, at 92 
minutes (Becker, et al., 2012). Table 3 shows the average time invested by the 
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teams in each design activity in this study and the average time invested by 
individuals in the previous Becker and Mentzer study. 

Information Gathering. Teams spent nearly twice the percentage of time 
engaged in gathering information. Information gathering was coded when 
students were actively requesting, reading, and reviewing information related to 
the problem or solution. Information requests could be made of the 
administrator. Teams working on the hallway traffic design challenge were 
provided with a laptop and Internet access. Student use of the Internet was 
generally coded as information gathering and represents a difference from the 
data collection protocol used with individuals and playground design teams as 
they did not have access to the computer. 

Table 3 

Mean and Standard Deviation Summary’ Statistics for High School Student Teams and 

Individuals 


Design Process Measures 

Individual 

(n=59) 

(Becker, et al., 2012) 

Teams 

(n=17) 



Minutes (SD) 

Percent 
of time 

Minutes (SD) 

Percent 
of time 

Total Time 

91.7(47.4) 


101.7(18.43) 


Problem Scope 

15.5 

18.0 

27.2 

26.3 

Problem Def. 

5.6 (3.1) 

7.7 

6.8 (5.49) 

6.7 

Info. Gath. 

9.9(13.3) 

10.3 

20.4(12.19) 

19.7 

Solutions 

63.2 

70.5 

55.6 

55.1 

Generating 

2.9 (6.6) 

3.9 

2.8 (1.62) 

2.9 

Modeling 

54.4 (35.4) 

60.2 

44.2 (13.29) 

43.4 

Feasibility 

4.4 (4.1) 

5.4 

8.0 (4.41) 

8.3 

Evaluation 

1.1 (3.5) 

1.0 

0.5 (0.78) 

0.5 

Realization 

8.2 

7.6 

24.2 

23.6 

Decision 

0.4 (0.7) 

0.4 

0.2 (0.32) 

0.2 

Comm. 

7.8 (13.0) 

7.2 

24.0 (12.87) 

23.4 

Other 

3.1 

3.8 

9.1 

9.0 


Modeling and Communication. Time allocated to modeling and 
communication show differences between teams and individuals. The teams 
tended to spend less time modeling and more time communicating. Modeling 
was defined as detailing how to build something, including calculations, 
estimations, determining locations, and description of how something will be 
assembled or fabricated. Communicating was defined as the efforts involved in 
telling someone how to build the playground. Communication efforts focused on 
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sharing the team’s plan with others and could be directed toward a contractor or 
a board of directors considering the team’s proposal. 

Playground vs. Pedestrian Flow problems. The playground problem had 
been used extensively in previous studies described above, but was potentially 
less relevant and authentic to high school aged students who are generally too 
old to use playgrounds and too young to have children of their own playing on 
playgrounds. The pedestrian flow problem was experienced daily by students 
and differences in design process times were evident in this study. Table 4 
summarizes time allocations as a comparison between problems. 

Problem Definition. Time spent on problem definition differed between 
the pedestrian flow and playground problems. Teams spent more than twice the 
amount of time reading, reflecting on and considering the problem for the 
playground compared to the pedestrian flow challenge. The playground problem 
presentation was longer and more specific. Constraints and criteria were 
specified as compared to the pedestrian flow in which constraints and criteria 
were not specified. The research team had anticipated that the lack of definition 
would permit students to develop their own constraints and criteria relative to 
their local problem, but, time in the problem definition phase was actually less 
when constraints and criteria were not provided. 

Information Gathering. Teams working on the pedestrian flow problem 
spent about twice the percentage of time searching for and digesting information 
relative to the problem than did the teams working on the less familiar 
playground problem. Examples of information gathered in the playground 
problem focused on identifying typical components on playgrounds such as 
swings, slides, monkey bars, and material characteristics such as strength, 
durability, and cost. Information across both problems included benchmarking, 
but on different conceptual levels. Searching for playground components was a 
concrete task resulting in a list of typical play things while the hallway problem 
yielded much more complex transfer from other schools or public places where 
traffic congestion was a problem. Students looked at airports as examples of 
moving people in short periods of time as a potential method of benchmarking 
and gathered these examples to spawn ideas in their scenario. The transfer from 
an airport or mall hallway to a school hallway was challenging for students 
perhaps because the population of users was different (i.e. adults in airports vs. 
students; adults may be motivated to run to their next flight vs. students who 
may not be interested in getting to the next class). 

Students in the hallway problem appeared to spend more time searching, 
perhaps motivated by their personal interest. In each administration of the 
problem, students were obviously bothered by the problem and were quick to 
engage as compared to the playground problem where students engaged at our 
request but seemed less intrinsically motivated. The hallway challenge included 
Internet access, which could have related to the additional search time. Our 
informal observations seemed to indicate that students not only accessed the 
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computer, they also gathered information from memories of direct observations. 
They recalled their experiences in airports and malls with pedestrian congestion. 
They recalled traffic flow rates and locker placements in the school and 
considered the impacts of this information on their design process. 

Feasibility. Teams on the pedestrian flow problem spent more than twice 
the percentage of time considering feasibility of their solutions as compared to 
the teams on the playground problem. Feasibility was defined as considering the 
practicality or viability of a solution or element of the solution. This was 
differentiated from evaluation in that evaluation included comparing two or 
more options while feasibility was passing judgment on one potential idea. In 
the playground problem, feasibility typically centered around cost in addressing 
the question: “Would an item/component cost too much?” Also, playground 
design teams considered the extent to which they met the constraints. This 
differed in the hallway problem because students were not provided with 
constraints or criteria nor did teams spend time to specify either constraints or 
criteria for the solution of the problem. Feasibility, however, consumed a much 
greater percentage of time as students attempted to determine if their solutions 
would work. Students implicitly must have identified some constraints and 
criteria as they talked about feasibility but not directly. Typical examples 
included students discussing the financial cost of a solution or the impact that 
the proposed solution might have on the problem without explicitly identifying a 
budget or rationale that costs should be limited or minimal. Some solutions were 
structural while others were behavioral and students considered their potential 
solutions and students’ behavioral responses. This led to discussions of teachers’ 
roles, administrators’ media campaigns for pedestrian traffic patterns and the 
feasibility consideration: “would it work?” 
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Table 4 

Mean and Standard Deviation Summary’ Statistics for High School Student Teams in the 
Playground Problem and Pedestrian Flow Problem 


Design Process Measures 

Playground Teams 
(n=8) 

Pedestrian Flow Teams 
(n=9) 


Minutes (SD) 

Percent of 
time 

Minutes (SD) 

Percent 
of time 

Total Time 

Problem Scope 

108 (10.9) 

25.3 

23.0 

95.8 (21.5) 

28.9 

29.4 

Problem Def. 

9.9 (3.7) 

9.2 

4.1 (5.4) 

4.4 

Info. Gath. 

15.4 (9.0) 

13.7 

24.8 (12.9) 

25.0 

Solutions 

62.4 

57.8 

49.5 

52.6 

Generating 

2.7 (1.9) 

2.5 

2.9 (1.4) 

3.2 

Modeling 

53.2 (10.0) 

49.4 

36.1 (10.3) 

38.1 

Feasibility 

5.4 (1.7) 

4.9 

10.4 (4.7) 

11.3 

Evaluation 

1.1 (0.8) 

1.0 

<0.1 (<0.1) 

<0.1 

Realization 

29.6 

27.9 

19.4 

19.8 

Decision 

0.5 (0.3) 

0.5 

<0.1 (<0.1) 

<0.1 

Comm. 

29.2(11.4) 

27.4 

19.4(12.4) 

19.8 

Other 

9.7 

9.1 

8.5 

9.2 


Feasibility seemed more relevant for students to consider in the hallway 
challenge as compared to the playground problem. Students were familiar with 
the hallway issues and the solutions had direct impact on their lives. They 
seemed to have capacity for understanding the complexities of hallway traffic 
more than complexities in the playground problem. Student design teams 
seldom considered the issues of safety in the playground, overlooking such facts 
as the difficulties that 2 year olds, have in negotiating ladders. If the students 
had been parents of children for whom the playground were being designed, 
they might have considered safety and functionality with greater understanding, 
but as 17 and 18 year olds, they seemed to lack a sense of understanding about 
the functionality and dangers surrounding playground equipment design. 

Team Gender Composition. Teams were comprised of single gender and 
mixed gender. Four all-female teams, seven all-male teams and five mixed 
gender teams displayed differences in design processes. The gendered nature of 
the teams was difficult for the research team to control and, as a result, did not 
split equally across design problems. Three of the four all-female teams were 
provided with the pedestrian flow problem while only one was provided the 
playground problem. Five of the seven male only teams received the playground 
problem while two teams received the pedestrian flow problem. Two of the five 
mixed gender teams engaged in the playground problem while three attempted 
the pedestrian flow problem. Results of teamwork disaggregated by gender are 
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presented in Table 5. Total design time varied across groups with female only 
teams finishing their work nearly 22 minutes before their all male counterparts. 
Mixed groups of males and females averaged about eight minutes less than all 
male teams. 

Problem Definition. While female only teams spent less time engaged in 
design, they spent more time on problem definition (8.4 minutes) than did the 
all-male (7.0 minutes) and mixed gender groups (5.2 minutes). This finding is 
particularly noteworthy, as most of the female teams engaged in the pedestrian 
flow problem which, according to data presented in Table 3, drew relatively 
little attention to problem definition (4.1 minutes). 


Table 5 

Mean and Standard Deviation Summary Statistics for High School Student Teams by 
Gender Composition 


Design Process 
Measures 

Female Only 

( n= 4) 

Male Only 
(n=8) 

Mixed Gender 
(n=5) 


Minutes 

Percent 

Minutes 

Percent 

Minutes 

Percent 


(SD) 

of time 

(SD) 

of time 

(SD) 

of time 

Total Time 

87.1 (13.3) 


108.8 (7.9) 


101.9 (25.8) 


Problem Scope 

26.6 

29.9 

27.1 

24.6 

27.9 

26.3 

Problem Dcf. 

8.4 (6.4) 

9.0 

7.0 (5.4) 

6.4 

5.2 (4.2) 

5.3 

Info. Gath. 

18.2(14.9) 

20.9 

20.0(9.7) 

18.2 

22.7(12.9) 

21.1 

Solutions 

44.5 

50.8 

58.4 

54.0 

59.9 

60.2 

Generating 

2.4 (0.6) 

2.7 

2.9 (1.7) 

2.6 

3.1 (2.0) 

3.4 

Modeling 

35.6(13.8) 

40.6 

47.2 (11.4) 

43.7 

46.2(12.8) 

45.3 

Feasibility 

6.3(1.8) 

7.4 

7.8 (4.4) 

7.2 

9.8 (5.2) 

10.7 

Evaluation 

0.2 (0.3) 

0.2 

0.6 (0.8) 

0.5 

0.8 (0.9) 

0.8 

Realization 

16.5 

19.6 

29.6 

27.3 

21.9 

20.9 

Decision 

0.2 (0.3) 

0.2 

0.2 (0.3) 

0.2 

0.3 (0.4) 

0.3 

Comm. 

16.3 (3.8) 

19.4 

29.4(12.7) 

27.1 

21.5(14.0) 

20.6 

Other 

9.8 

11.3 

10.2 

9.4 

6.7 

7.0 


Modeling and Communication. Most of the differences in overall design 
time were related to modeling and communication. Females spent about 25% 
less time modeling than did males and mixed gender groups. The average 
female modeling time was 36 minutes while males and mixed groups spent 47 
and 46 minutes respectively. Male student teams spent nearly twice the amount 
of time communicating (29 minutes) as did female student teams (16 minutes). 
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Discussion and Implications 

Discussion focuses on four topics: inter-rater reliability, problem definition, 
evaluation and decision making, and modeling and communication. The 
development of this study was based on a series of studies situated on the 
college level (Atman et al., 2007). In these studies, experts were provided with a 
design problem as a comparison group to college students. These experts 
provided a verbal protocol individually and were provided with a very 
comparable experience engaging with the playground problem as were 
participants in the current study. Overall, experts spent more time engaged in the 
problem with nearly 132 minutes being their average as compared to teams of 
high school students whom spent nearly 102 minutes. This difference in time 
was primarily in modeling and communication. Experts spent more than 55% of 
their time (about 73 minutes) modeling while teams of high school students 
spent less time modeling (about 43% or 44 minutes). However, high school 
students spent more time communicating (about 23% or 24 minutes) than did 
experts who allocated about 5 minutes or 4% of their time. 

Inter-Rater Reliability. Reliability of modeling and feasibility coding was 
low despite extensive efforts to calibrate by the research team. Some of the lack 
of agreement could be related to researcher calibration, but the research team 
suspected that the lack of agreement is also related to lack of clarity by the 
student teams about the nature of these activities. The students were vague about 
how modeling was related to other aspects of the design process, particularly 
communication. What initially appeared to be graphical sketching as a method 
of developing ideas and laying out a potential solution for discussion evolved 
into a document for the final proposed solution. Differentiating between 
modeling and communication in the abstract was simple for the undergraduate 
research assistants as the difference centered on purpose. If the purpose of the 
sketching, for example, was to understand and improve appearance, 
functionality or fabrication techniques and the team used this information to 
think through challenges and determine specifications, it was coded as 
modeling. If, on the other hand, team effort was directed at documenting their 
plans for fabrication for the purpose of telling someone how to build from the 
plans, it was coded as communicating. Student teams often started modeling and 
the work evolved into communication. This evolution made precise 
determination difficult. In cases where the transition was gradual and vague, the 
coders generally defaulted to modeling until there was evidence that the purpose 
was an attempt to communicate team intentions/plans. In some cases, teams 
were very deliberate about this transition. In other cases, the transitions occurred 
gradually but were clarified later. For example, what might have appeared to be 
modeling was later determined to be communication and codes were changed 
appropriately as the coding process progressed through the team’s work. 

The boundaries between thinking on paper via sketching and making notes, 
and communicating with external stakeholders blurred and presented the 
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research team with difficulty identifying student intentions. At the end of the 
design challenge, students frequently presented rough sketches and messy notes, 
resulting in poor quality technical communication. If this situation is to be 
improved, teachers should make clear to their students how to communicate 
technical information in a persuasive way to external stakeholders. Classroom 
experiences might be focused on presentation skills where students present their 
work to other classes or an invited audience of people who are not familiar with 
the daily student design experiences. This external audience would challenge 
students to provide details and rationale for decisions made in context which 
was generally absent from student work in this research. 

Problem Definition. Students engaged in dialog about the feasibility of 
their potential solutions or elements of solutions in the hallway problem more 
than in the playground problem, but problem definition was considered more in 
the playground problem. These two activities may be inversely correlated such 
that a general lack of problem definition would lead to a tendency for students to 
be quick to question whether a potential solution would work. In the hallway 
problem, student teams seldom made explicit the constraints and criteria which 
made determining the feasibility of an idea more difficult and time consuming. 
The National Center for Engineering and Technology Education suggested, “As 
designs are considered for viability, optimization is essential. Students should 
make their value structures and goals for design success explicit early in the 
decision process. This sense of clarity provides opportunities to select and 
promote designs that make the most successful balance of trade-offs” (2012, p. 
26). In the less structured hallway problem, students would consider an idea and 
then ask if it would work or be too expensive without having specified the 
definition of success or budget. 

Teachers should encourage students to identify the constraints and criteria 
as well as how success should be measured early in the design process. 
Feasibility considerations were slightly higher for team based design problems 
than individual design problems. Further research might test for a causal 
relationship, which if present, would indicate that teamwork might facilitate 
experience and exposure to critical thinking about solutions in the feasibility 
phase of the design experience. The National Center for Engineering and 
Technology Education caucus of 2012 suggested that “In collective team efforts, 
students may hold each other accountable for meeting criteria” (p. 18). The 
sense of accountability may have manifested in feasibility as students questioned 
each other’s ideas prompting consideration of flaws and opening the door for 
improvements. Stakeholder interests were included in student discussions of 
feasibility in the hallway problem much more frequently than the playground 
problem. In the hallway problem, they mentioned considerations such as how 
students would interact with their solution, how teachers would be involved and 
react to students in a redesigned hallway. They considered impacts of hallway 
reconstruction on the neighboring rooms and how the changes impact roles of 
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librarians, cafeteria staff, and classroom teachers. Occasionally, parents and 
shopkeepers were mentioned in the playground problem as stakeholders but with 
far less emphasis. This may be related to the sense of relevance provided by the 
hallway problem, because students cared in a very personal way about the 
success of their design and considered a larger system of stakeholders. 

Evaluation and Decision Making. In both design problems, evaluation and 
decision making activities were rarely observed. Student teams spent very little 
time comparing alternatives on a criterion, which was our working definition of 
evaluation. Students also spent very little time choosing among the alternatives. 
Decision making was defined in this study as a deliberate choice between two or 
more alternatives. A typical decision and evaluation activity in the playground 
problem included material selection. Student teams would ponder using wood or 
metal as a construction material, discuss costs, strength and durability, then 
make a selection. In the hallway problem, even fewer evaluations or decisions 
were observed. This lack of evaluation and decision making may be directly 
related to the fact that students developed few alternative solutions during the 
brainstorming phase. While they did brainstorm and develop ideas, selection 
decisions tended to be related to the feasibility of individual components of the 
solution rather than a comparison of alternative solutions. Students would 
frequently say, “Let’s put in a slide, it’s cheap” with no externalized comparison 
of the alternatives. 

The general lack of evaluation and decision making may relate to a lack of 
alternatives for consideration. Students tended to think about new ideas until 
they had a few viable options and developed those into their final design. The 
lack of alternatives generally reduced the need to evaluate differences between 
them and reduced the number of decisions (choices between alternatives) to 
make. Teachers should encourage students to develop a significant list of 
alternative ideas before evaluation and decision making. Decisions regarding 
materials were made on the playground problem, but in both design problems, 
students did not develop many alternative designs. They considered the 
advantages of different materials but seldom considered holistically different 
solutions. 


Future Research 

Findings from this study suggest potential trends and correlations between 
individual design activities and group design work as well as suggesting 
differences between teamwork on two different kinds of problems. This study 
had a sample size of 17 teams in total. Eight teams were challenged with the 
playgroup problem and nine teams engaged in the hallway problem. With only 
eight or nine teams in comparison, statistical analysis was not conducted. This 
work is potentially foundational to larger studies as it may allude to trends and 
correlations that could be tested in experimental or quasi-experimental research 
conditions on a larger scale. 
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This work may have implications for methods of group based verbal 
protocol studies. Future research efforts using student observations might 
increase their inter-rater reliability measures with teams by being able to identify 
what papers students are using when they are writing or sketching. In work with 
individuals, reviewing the digitized artifacts and video observation data typically 
provided ample evidence to determine what students were writing. However, 
with teams of students and multiple artifacts, researchers were less able to 
identify which paper was being used during a particular phase of the design 
process. Cameras positioned from an angle overhead might allow association 
between papers and content of the writing. However, in this research effort, a 
wide angle video of four students made identifying what was written and when 
difficult. In addition, the research team noticed that they were able to code 
feasibility, for example, consistently, but they had difficulty determining exact 
start and stop times. One research assistant might include a background 
statement as a lead in to feasibility while another might code a narrower band of 
feasibility leading to general agreement between researchers but low Kappa 
values. 

From a methodological perspective, the inverse relationship between time 
spent in problem definition and time spent in feasibility might provide insight 
into student thinking about problem definition. Though problem definition in the 
hallway design task was seldom coded, future researchers could use feasibility 
as a method of extracting student definitions of the problem. Feasibility 
considerations have inherent value statements that could provide a proxy for 
problem definition. Therefore, by analyzing student conversation about the 
feasibility of an idea, a future research team may be able to identify the implicit 
constraints and criteria that students do not mention explicitly. For example, if a 
student judges a potential solution to be too expensive, we can infer that cost is a 
criterion even though it was not mentioned as one. Students mentioned concerns 
such as slide or platform height or soccer field location in the playground 
problem, which relate to the constraint presented that the playground must be 
“safe”. In this example, the students silently operationalized safety by 
considering a minimum distance from a nearby road to the soccer field. In 
addition, students’ brainstorming activities may provide insight to the problem 
definition in that they tend to think of potential solutions and the commonalities 
across those solutions may be hints into their problem definitions. As an 
example, students who list different ways of controlling student hallway traffic 
such as traffic lights, teachers, or mirrors, may suggest implicitly that widening 
the hallway is not practical or that they feel constrained by the lack of resources 
to make major structural changes in the school architecture. 

Further research might investigate qualitative differences in the ways and 
methods in which all female teams engaged in problem definition as compared 
to males and mixed groups. Female groups spent more time and a much higher 
percentage of time on problem definition than their peer groups. It might prove 
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beneficial to practitioners to understand the impact of this additional time on the 
design thinking. Does this additional effort represent a deeper understanding, 
and does a deeper understand cause improved design performance and more 
effective solution development? 

Additional investigation may clarify differences between gender and 
modeling and communication activities. All female groups spent dramatically 
less time modeling and communicating which accounted for most of the 
differences in overall design problem time. Are females modeling and 
communicating less or differently than their male only and mixed gender team 
counterparts? Teachers should consider and make explicit modeling and 
communication expectations so that group efforts are comparable. If the teacher 
is expecting significant modeling and communication activity, the teacher 
should be sensitive to the possibility that team gender composition may impact 
these behaviors. Teachers need to monitor these activities and clearly articulate 
the importance and purpose of these elements of design so that teams allocate 
time where the teacher is expecting. 

The National Center for Engineering and Technology Education suggested 
characteristics appropriate for engineering design challenges for high school 
students (National Center for Engineering and Technology Education, 2012). 
This study used the characteristics to adopt an instrument used in previous work. 
Elowever, relevant design problems might not only improve design challenges as 
measurement instruments but might also fundamentally change the learning and 
teaching process. Data were not gathered in this study about the typical 
classroom design problems that were the foundation of student experience. 
Future research could investigate impacts on student learning related to 
authenticity and its impact on motivation and engagement. Students who are 
actively engaged in learning may interact with the design problem very 
differently and therefore their learning and experiences may be different from 
student less engaged. In addition, while most students in this research were 
seniors and each had taken numerous design related courses, the number of 
design problems and their duration were not studied but could impact student 
performance. Further study might investigate classrooms that conduct design 
problems regularly as a pedagogical approach as compared to classrooms which 
situate design problems only as a capstone experience. 
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Summary 

This study provided seventeen teams, each comprised of 2-4 high school 
students, with a team based engineering design challenge. Observational 
protocol analysis was conducted based on a foundation of previous work, 
including the adoption of previous coding schemes. Differences between groups 
and individuals were compared. Teams of students were split in two groups; one 
set of teams received a playground design problem while the other received a 
hallway design problem. Teams worked up to two hours after school on the 
design problems and provided the recommendations resulting from their work at 
the conclusion of the session. 

This material is based upon work supported by the National Science Foundation 
under Grant No. 0426421. Any opinions, findings, and conclusions of 
recommendations expressed in this material are those of the authors and do not 
necessarily reflect the views of the National Science Foundation. 
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